Repository navigation
Open DictStore/TreeStore files written by python-blosc2 <= 4.0.0 again - #788
Open
FabianIsensee wants to merge 1 commit into
Open
FabianIsensee wants to merge 1 commit into
FabianIsensee wants to merge 1 commit into
Conversation
Up to 4.0.0, the embed.b2e of a DictStore or TreeStore carried only the "b2embed" marker of an EmbedStore; the b2dict and b2tree markers came in 4.1.0 (66b1c58), together with the meta-first dispatch in blosc2.open(). That dispatch reads the marker from the embed.b2e inside a .b2d or .b2z and, seeing "b2embed", opens the whole store path as an EmbedStore, which fails with "blosc2_schunk_open_offset(...) returned NULL". Old .b2d stores have failed this way since 4.1.0; old .b2z stores since 4.8.1, when 92e0233 made the zip probe work (it used to call the module-level open() and always fell back to the extension). "b2embed" therefore does not identify an EmbedStore, so drop that branch: .b2e files reach the same EmbedStore call through the extension dispatch, and old stores open as they did in 4.0.0 (.b2z as TreeStore, .b2d as DictStore). Stores written by 4.1.0 or newer still dispatch on their b2dict/b2tree marker. The explicit DictStore()/TreeStore() constructors were never affected. The test fixture is a DictStore written by python-blosc2 4.0.0 (2 kB). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DiPFzu8ziFQsxrJUkHCrzx
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
blosc2.open()fails on.b2dand.b2zstores written by python-blosc2 4.0.0 or older withRuntimeError: blosc2_schunk_open_offset('<path>', 0) returned NULL:.b2dsince 4.1.0,.b2zsince 4.8.1.DictStore(path)/TreeStore(path)still open them. To reproduce: with blosc2 4.0.0,with blosc2.DictStore("old.b2z", mode="w") as s: s["/a"] = np.arange(10); then with 4.14.1 or main,blosc2.open("old.b2z", mode="r")raises (blosc2.DictStore("old.b2z", mode="r")works).Cause: up to 4.0.0, DictStore and TreeStore wrote only the EmbedStore marker
{"b2embed": ...}into theirembed.b2e;b2dict/b2treecame with 66b1c58 (4.1.0), which also made_open_special_store()dispatch on that marker before the extension. For an old store the marker isb2embed, so the whole.b2d/.b2zpath was opened as an EmbedStore. For.b2zthe zip probe in_meta_from_store()only started to work in 92e0233 (4.8.1,builtins.openinstead of the shadowedopen), which is why those broke later.The fix drops the
b2embedbranch from_open_special_store(): that marker does not identify an EmbedStore, since old stores carry it too. A.b2efile reaches the sameEmbedStore(urlpath, mode=mode, deserialize=deserialize, **kwargs)call through_store_from_extension()(same offset check and error), and old stores open as they did in 4.0.0 (.b2zas TreeStore,.b2das DictStore; old files have no marker that tells the two apart). Stores written by 4.1.0 or newer still dispatch onb2dict/b2tree, andb2remote_storeis untouched. This also changesblosc2.open(old_store, mode="w"): on main it replaced the old.b2z(and even a.b2ddirectory) with a single EmbedStore frame file that no longer opened; now it starts a DictStore/TreeStore, as for newer stores.Test:
test_open_store_written_by_4_0_0intests/test_dict_store.pyopens a DictStore written by 4.0.0 (new fixturetests/data/legacy_store_v4.0.0/dictstore.b2z, 2 kB: an embedded leaf, an external leaf and a nested key) as.b2zand unzipped as.b2d, withblosc2.open()in the default mode, in mode "a" (which writes a.b2zback on close) and again, and checks the class and the data. Both cases fail on main. Full suite: same results as main apart from the new test. Outside the suite,blosc2.open()opens all 105 store files written by 21 releases (3.7.0 to main: DictStore/TreeStore as.b2z/.b2d, EmbedStore.b2e) with the right data, and every file written by 4.1.0 or newer opens as the same class as on main.I used Claude Code (Anthropic) to bisect the regression over the PyPI releases, write the fix and test, and draft this description; I reviewed the change.
🤖 Generated with Claude Code